— 5^ 

DOCIMENT RESUME 



ED 244 437 

AUTHOR 
TITLE 



institution 
spons agency 
rep6rt_no 

pyB_PATE 
CONTRACT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



EC 162 450 

Skiba> Rusgell; Dehp ^ ' Stanley L . 
A Correlational Analysis of the Statistical 
Properties of Titne-Ser ies Data and Their Relationship 
to. Student, Achievinent in Resource Classrooms • 
Hinnesota_Un i v . ^ Minneapol i s • Inst, for Research on 

Learning Di sab i li t ies . ^ . 

Special Education Programs. (ED/bSERS), Washington, 
DC._ 

i RLD-RR-i38 
Sep 83 
300-80-0622 
39p. 

Reports - Research/technical (143) 
MF01/Pe:02 Plus Postage. 

Elementary Education ; *Learn ing Disabilities; 
^Measurement Techniques; Research Methodology; 
Statistical Distributions; Student Evaluation 



ABSTRACT ^ ^ 

Resource room students (N-6B) in grades 1-7 of three 
rural and suburban Minnesota school districts participated in a study 
of time series data generated by a curriculum-based measurement 
system. A principal compohehts factor analysis wis performed to 
summarize relationships among the time-series propertied and 
properties of 'the measurement system. In addition ^ multiplje 
regression analyses were used to identify the relationship of such 
variables to achievement. Results indicated that the statistical^, 
characteristics of time-series data are not necessarily independent 

in naturally occurring data^ and can be used in predicting 

achievement. Findings had implications in trai^^ng practitioners in 
the. use and interpretation of measurement systems based on 
tim^-series data. (Authpr/CL) 
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Abstracr\ 

: % : 

Research has shown that statistical properties of time series 
data can influence visual inference based on those data^ little 
research, however, has investigated - the ^ interactions of these 
properties in 'experimentally generated data. Tn the current study, 
time series data for 68 students generated by a curriculum-based 
measurement system were analyzed.. 'A principal comporfents factor 
analysis wa? performed to summarize relationships 'among the time- 
series properties and properties of the Measurement system. ' In 
addition, multiple regression analyse^ were used to identify the 
relationship of such variables to achievement.^ Results i^ndicated that> 
the statistical characteristics of time-series ' data are hot 
necessarily independent ' iri naturally occurring data, and can be used 
in predicting achievement. Tmplicatidns for training practi tioners in 
visual inference are discussed. j 



A Correlational Analysis of the Statistical Properties of 
Time-Series Data and Their Relationship to Studeht 
Achievement in Resource Glassroems 
in recent year, considerable controversy Has been generated 
regarding the appropriate method for analyzing tiirie-ser ies data. 
Visual analysis of the data has been favored by many practitioners 
(Kratbchwi 1 1 , 1978). Those who favor visual inference argue that it 
i; increases the probability that only large, practically significant 
changes will be detected (Parsonson S Baer, 1978). Advocates of 
solely visual analysis also have asserted that reliance on post-hoc 
statistical procedures may lead researchers to ignore the necessity of 

maintaining strict experimental control (Michael, 1974), and may 

\ - . 

reduce this general izabil ity and repl icabi 1 1 ty of experiments. 

On the- other hand, it has been argued that the statistical 

characteristics, of time-series data make them too complex for reliable 

visual analysis. Jones, Weinrott, and Vaught (1978) have shown that 

sma-l 1 changes that are statistically significant may net be detected 

;by visual inspection of the graphed data. These riondramatic but 

reliable ef f ects-^may, in fact, prove clinically significant under some 

- ' _-- 

conditions (Kazdin, 1976). _ Final ly, 'un-ique properties of time-serie^s 

data, such as -serial depencfehcy fthe tendency for successive dat^ 

-points to be correlated and thus ndniridepehdent);^ may cause decreases 

in the reliability of visu.al analysi?. Jones et al . (1973) found that 

under conditions of serial dependency^, inter-observer ^agreement using 

Visual analysis was only .39. . ^ 

) 

Id addition to serial dependency, other properties of time-series 
data may*" affect the nature of the conclusions drawn from such data. 

1 
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' Jones, Vaught, and Meinrott (1977) noted tfrat judgments made about the 

/' impact of an intervention on time-series data codid be influenced by 

stab.ility bf baseline behavioral scores, variability the data 

within and across phases, number of data points, and changes in level 

of performance. Kazdin (1976) identified the overall trend or slope 

of the data, and changes in that trend, as another important 

consideration. . 

Furlong and Wampbld (1982) . investigated the effects of various 

■ _ •* 

statistical properties bf time-serie§ data bri the judgffjjnts made about 

those data. They generated graphs that varied along the dimensions of 

level, trend, scaling, and variatibn. Ten reviewers, members of the 

Journal bf Applied Behavibr Analysis editorial board, were, instructed 



tb sbrt the graphs into as many categories as they wished. Analyzing 

: ' ' 

the reviewers' sorting strategies, the invi^jpti gators " concluded th^t 

the maabrity of experts classified the time-series data aeeardirrg to 
comrnon intervention patterns. Only a minority appeared to take 
variability into account when making visual inferences. Further, 
rev.iewers who attended on1/\to the absolute size bf iriterveritibri 
effects classified the'e^ect patterns less adequately. 

In a similar study," DeProspero and ^Gohen (1979) generated time- 
series of the "ABAB reversal'' type, and sent them tb- 250 reviewers of 
behavioral ^journals for judgments on degree bf experimental control 
expressed by the data. Visual irispectibn bf the data resulted in a 
mean inter-rater agreement of .61 between pairs bf judges assigned the 
same graph. A mean shift in the data pattern consistent with, the 
hypothesized effect bf the experimerital variables was the most 
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^mpdrtaht irif l^Uehce on' visual inferehce, Variabi 1 1 ty was attended to 
less welK Noting the wide ranqe of opinioi^s concerning tha important 
characteristits oF any g>iven 'data display;' dePrgspgro ^d Cohen 
concluded that a behavioral researcher relying on visual analysis 

"would not be likely tp get . the . same answer twice" (p. 578) and 

: -/ __ * 

recommended supplementing ■ any'; visual analysis with statistical 

< : . • ■ ' 

analysis. 

^* _ • _ 

' Tindal, Beno, and Ysseldyke {1§83) Investigated tf»e effects of 

- jf--- 

N/arious combinations of slope and variability on reliab-ility of visual 

^ ^ ' > _ r - 

judgments, using classroom teachers as subjects. The teachers^/were 

. . ' . _ _ . _ _ _ _ _ <. 

trained in teehniques of visual inference, then asked to determine thS 

* _ - - 

effectiveness of the'program depicted bh time series varying along the 

dimensions Qf slope and variability. The re'§,ults- indicated that slope 

and variability influenced visual judgments, both singly and in 

ihteractidh. Reliability of visual inference was influenced by 

variability, arid decreased ^mdst when^ variabi 1 i^ty increased between 

phases* Tri addition^ the use of aimlines as^^an aid irrvisual analysis 

» 

did not seem to have an effect on the reliability of visual ^halysi:^. 

<• 

^ Research thus seems to Indicate that srtatistical. parameters do 
-influence both the types, and the reliability, of judgments , made 
through visual analysis, of time-series data. Decisions made on the 
basis of visual inference seem most influenced by change in the mean 
leveT^of performance and trend, wh-rle variability is attended to less 
well. Yet variability seems to have a major impact on'the reliability 
of visual analysis.: 

While- research has , begun to ihvesti gate some of the statistical 
parameters that influence time-series analysis, many areas remain 
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unexplored. All df the research to date has used' artificially 
generated time series, randomly varying the data characteristics under 
investigation. Y<5t, it is possibl? that the statistical properties of 
time-.series data, .suph, as slope, variability, arrd trend, covary in a 
nonrandbirf fashion in data generated in experimental, c1iriica1,^or 
classroom situations. ; Tf this is indeed the case, it may be •important 
to tr^in teachers, and others' who will make decisions based on, the 
data, to attend to these characteristics and their interaction. . In 
addition, little attention has been given to possible relatibtlships 
betweetT'sueli characteristics arid long-term dUtcbmes, such as student f 
achievement. .Tt may well be that vari abi 1 i ty pr riumber df data points 
are as important iri predictirig butcbme as are more attended to 

attributes, such as trend or level.- 

— ' __ _ . _ _ _ j»/ - _ _ * _ . 

The purpbse of the current study was to investigate the 

statistical properties of ^ime-series data generated by a curriculum 

^ based measurement system in reading. The use of actual student data 

.^allowed exploration of the relationships among various' characteristics 

df the data. In addition, the current study was designed to determine 

^whether properties of a system based on time-series data analysis can 

be used Ib predicting reading achievement. ' 

S ub i ects * 

The subjects were 58 resbdrce room studerits iri three rural ari^ 
-suburban Minnesota school districts. AH subjects t^ere participants 
in research on the effects of teachers usirig frequerit curriculum-based 
measures of student performance. Subjects ranged in grade placemen^ 



-from first to seventh grade; the. fHstr ibutidh of students by grade 
level is shown in Table 1. ( . ' 



Insert Table 1 about, h«re 

Al 1 students were r&be i ving some resource room .instruction and 
had been receiving such special instruction for anywhere from a few 
months to. six years fX = 1,96 years'*. The time spent in reading 
instruction- in the resource room ranged from 15 minutes to 105 minutes 
per day, with a mean of .46 minutes per day. The students' teachers 
averaged two years teaching'exper ience in regular education, and five 
years in special education. . ^ 

Measures 

The independent vari/able for all analyses, student performance on 

a curri.culum base^ time-series measurement system, was measured 

through the^statistical properties of that system, described in the 

Procedures section. The dependent measures were two measures of 

achievement: timed reading samples from _^hree third grade passages 
• r"", ^ 

(Deno^ Mirkin^ S Chiang, 1982) and four subtests of the Stanford 
Diagnostic Reading Test fSDRT). . ^ ^ 

Ach lavement measures -At thre^ different points in time during 
the study, three one-minute oral reading measures^ consisting of 
randomly selected passages from the third grade level in Ginn 720, 



were admin istered to the studen " , These measures were sel ected 
beca.use of their techhical adequacy (Deno et^al., ?982) and 
sensitivity to change (Marston, Lowry, Deno,^ Mirkin, 1981). These 



6 

^ curriculum-ba^ed measures had heon fotjhd to bo as reliable arid valid 

_ ■ ■ _ _ _ - - ■■ ■ - - - ( t- ■ _ _ . 

'as traditional standardized testsj yet moro 1 ikely to reflect small 

changes in performanct^ . The measiiremehts were conducted by directing 

the student to begin reading at the top of the page and continue 

reading for one minHte, at which time the examiner would say stop. Tf 

the student came to a word he/she did hot know, the examiner would 

supply the word and prompt the reader to continue. While the student* 

was reading, the examiner followed along on a copy of the passage and 
♦ 

marked errors of •,sUbsti tutioh arid dmi"ssidh. Following the reading, 
the numbers of words read correct and incorrect were counted and 
recorded, with no feedback given to the student. These three reading 
measures were given at the beginning of the study (pretest), in the 
middle, and imrnediateVy at the end ^of the study. 
- ^ Two subtests from the Sta nford -tU ^ gnosUx Rla ding Test (Karlsen, 

Madden, S Gardner^ 1976) also were given as posttest measures. The 
Structural Analysis and Reading Gomprehension subtest^s were 
administ'ered along wit^ the reading passage measures. Each of the 
SORT subtests has two^ parts, witHi Structural Analysis focusing oh 
-syllabication (blendi ng • and division) and Reading nompreHensidii 
focusing on answering both literal and inferential ^questions »for 
previously- read passages-; ' . . 

P-roc edur^S- 

The resource room teachers were trained in the use of the 
measurement proe^dires during a series of three':,half-day workshops at 
the beginning t)f the school year: Training was based^on the manual, 
Broced-urfes" to Develop and- Monitor Progress on lEP Goals (Mirkin, Deno, 




Fuchs, -Wesson, Tlndal , Marstoh, ft KuehniG, 1981) The teaehers 
continued to use the- measures over the entire school ^year; Visits by 
observers in December, February, and: May, and frequent phone contacts, 
provided feedback to the teachers on the accura^^ of ' their 
implemehtat ion of the measures. 

Measurement consisted of bne-mihute timed samples dS reading from, 
the student's curriculum. Based dh the^'results df previous; research, 
the placement level fdr testing was set at a criteria of 20-29 words- 
cdrr^ect-per miniTte fdr grad(3S 1 and 2^ and 30-39 words correct per 
minute for grades 3^-7. Once this level was determined^ passages were 
chdseh randdmly .frdm the placement level textbook for measurement 
pjjrpdSGS. Measurements were conducted three to five times each week. 
Bdth number of words reM-^orrectly an-d number of errors in one minute 
were recorded, and plotted on an equa^ interval chart. Continuous 
graphed- resul ts allowed teachers to develop a visual recorS of student 
progress, similar to the one represented in Figure 1. 



Insert Figure 1 about here 



Teachers were instrae%ed to write TFP long-range gdals\ (LRG) 
using both the entry Ifvef criteria an(^a' desired year-end mastery 
criteria, usually 70 words cdrrect per minute with hd mdre than 7 
errors. Theformula used in writing the Idrig-rahge "goal is shown in 
Figure 2. \ : 
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Insert Figure '2 abdat here ■ " • . : 

*> ^ . 

_ > , , ■ . . . 

Short-term objectives were based on the ^ long-range goals Tn 

order to compute the short-term objective, teachers first 'isubt'racted^ 

the baseline level of performance from the^cri terion level listed \^ 

the LRG". hividing 'this difference by the number of weeks until, the 

annual ' review, the^ arrived at the number of words per week , gaiH- 

necessary to meet the long-rajge goal, criteria. Tffe' format used for 

writing. short-term objectives \% given in Figure 3. . ; ; ■ - • 

t" Insert Figure 3 about. here- . . : : : ' ■ ' • r 

In addition^ ' the teachers were trained at the : h^STi'^'^'^nq^af 
yeair, arid agairi at mid-year, in the use of the nieasUrement procedures ' 
for evaluation of the instructional program. . In or^der to niofrltor 
student growth, the baseline reading ley^l.and ihe - Tong-rang&::^>^^^^ 
were connected by an aimli'ne that show^^ the ^|tu desired 
progress. Every seven ct^ta points the . teachers w^r*e to; evaluate . 
student growth using a decision rule that required use of the guarter- 
intersect methiBd (White & Haring, 19801 to determine slooe. An . 
example is giveh in Figure 1. If the student was progressihg at ;a 
rate equivalent to or greater than that indicated by the aimline, the 
instructional program was continued; if the projected, rate of ^growth 
was Tess than that indicated by thb aimline, the teacher was tq mafce. a 
substantial change in the student* s program. 



la 
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student performahce data on the direct repeated measures were 
^dllected arid charted over a six^month period for the 68 resource room 
students. Variables generated from these graphed performance data for 
experimental analyses are discussed below. 

Desi qh ■ s ' 

The aihalyses used in the current study were correlational; 
Descriptive analyses' for all variables except steepness of the almlirie 
are presented elsewhere (Sklba, Marston, Wess'on, Sevcik, ^ Deno, 
1983). ^ ' . ^ 

To explore the relationships among the properti es ^_ of the data, a 
factor analysis was performed in order to sammari^e a large number of 
correlations. Variables included in the analysis were those variables 
available to the. teacher in a visual analysis of student time-series 
data. The fir-sf subset of variables' generated were those cdmmohly 
identified in time series literature as ^ influential in visual 
inference: level of performance, t^^end, variability, .and number of 
data 'pointy. Two measures of level of performance were included: the 
Y-intercept (Y-INT) as an estimate of baseline performance, and the 
mean level of performance for the year (MY).. Based on the six months 
of graphed perfbrmance data, the overall trend of the data was 
estimated by cdmputihg a regression slope for each individual (ShOPEh 
In- order to test the probability that the slope represented a 
significant trend in the student's reading performahce over time, and 
ndt^ ah artifact of random variability, the c-statisticl was applied to 
each individual time-series, as recommended by Tryon (198?). The z-^ 
.score of the c-statistjc (Z-SCORE)^ was used a/ an estimate of the 
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significance of the slope. Variability was measured by' the standard 
ernor of estimate (SEE) of the regressidri slope. Number of data 
points fDATAPTS) was measured in terms of number of measuremehts jDer 
week, since such a measure proved more comparable across teachers "than 
total hurtitjer data points (which was highly irifluericed ti^ begihnirig 
and end of year school schedules). 

In addition, a number of _ variables specific to the curriculum 
based measurement system 'employed^w^i^^ in the analysis, since 

these could be expected to influence the time series data pattern. 

The aimline (AIMLIM'E) was calculated using the formula for the short- 

• _ _ _ _ __ . _ __________ 

term objective, and served as an estimate of student progress 

necessary to reach the long-range goal from the Y-intercept. Student 
success in meeting the goal was estimated both in terms of whether the 
goaK^as met at least once (GOAL) and by the number of times the long- 
range goal was^ exceeded in the time series (NGOAL). The number of 



phase changes was measured by the number of instructional 
interventions fCHANGE) implemented over the course of the school year. 
In order to determine whether any. property or properties of the 

time-series data could be useful in predicting' reading achievement, a 

f _ 

series of regression analyses was performed with the achievement 
measures as dependent variables. The_ independent variables were the 
var i ablGS 1 is ted above : mean for the year, Y-intercept, number of 
data points peV week, slope, number of instructional changes^ standard 
error of estimate, z-scdre of c-statist^ic, wheth^^r the goaUwas met, 
number of times the goal was met, and steepness of the aimline. Since 
school aohievf?ment has been shown ^:o be correlated most highly with 
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entering student abil Ity (Bloom, 1976; Borg, 1980) , two methods of 
controlling for achievement were used. First, pretest achievement Tas 

measured by passage data) was forced as the first indeperirfen^ variable 

_ .. \ 

for a1 1 regress ion analyses conducted ^on the scores obtai ned during 

the th'ird dat^ collection. Second, two gain scores were ncq led 1ated: 

a score representing the absolute gain in' words re^6' per minute 

between the first and third timed passages, and the conversion 'of 

these absolute gain scores into percentage gain. Achievement was 

standardized by grade control for age effects (except for gain 

scores, which were based on raw data). 

Resul ts 

Results are presented in two sections.; First, the relationship 
among the variables was investigated by performing a factor analysis. 
Second, the relationship of these variables to student achievement was 
explored through regression analyses. Descriptive data for the 10 
variables examined are presented in Table 2. 

Insert Table 2 about here 



Relationships Amartq- J:Re— Variables- 

A principal components factor anal ysi s of the _time-ser ies 
variables was performed. Factors with eigenvalues equal to or 
exceeding 1.00 were rotated, and variables that correlated with the 
factors at a level pf .30 or greater were analyzed. The three factors 
retained accounted for E7T6% of the total variance. As indicated in 
Table 3, the itams that correlated most strongly with Factor 1 have'td 

i ie 
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do with leva! of perfbrmarice, such as the Y-ihtercept or the mean 
level of performance for the year. The standard error of estimate, 
and both goal-meeting variables, also . loaded moderately on this 
factor, while number of instructional intervehtions loaded moderately 
riegativeiy. Factor 1 accounted for 30.1^ of the variance. Factor 2,. 
explaining 23%- of the Variance, was most heavily^ influenced by the Z- 
scare, a measure of the signif-icance of the slope and the number of 
data points per week. The number of times the goal was met correlated 
moderately with this factor, while the Y-intercept and the standard 
error'df estimate loaded moderately negatively on Factor 2. Factor. 3 
could be conceptualized as a aiml ine/slope factor and accounted :for 
\^:A% of the total variance. The number of times the goal was met 
showed a moderate negative correlation withthis factor. 



Insert Table 3 about- here 



Reil^ti-anShip of Time-Series Variables to Achievement Variables 

; ResuKs of the regression of the tim^-series V|riables , on 
' ^ ' _ ■_ _ - _ _ 

achievement measures are presented in Table 4._ As can be noted in the 

second column, the number of times students met their qoal and the 
number of data points per week strongly predicted achievement on the 
majority of the achievement measures. Tt is interesting to note the 
increase in the proportion of residual variance explained by the time- 
series characteristics from the Time 2 passage scores to the Time 3 

/ _ I- _^ 

passage scores. .Mean level of reading performance for the year and 

the Y-intercept level are moderate predictors of overall gain on the 
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passages (Passage Gain Score). Ndmber of ehariges made in the 
students' iristructibrial "program negatively [Dredicted scores. on the' 
SORT cdmpreheris ion subtests and total score, while steepness of the 
aimlirie drawn by teachers at the beginaicrg of the ye^ar negatively 
predicted perfdrmahce on both final pas^sr^ge scores and isiverall gain in 
words read per minute. 



Insert Table 4 about here 



Discussion ' 

Previous analyses of computer generated time -'series have 
indicated the influence of , statistical properties of the data in 

visual inferences made regarding the data. Yet the majority of these 

^ _ _ _ ^ _ __ _ 

studies have explored ^the properties in isolation, seldom taking into 

account .interactions that may occur in experimentally generated data. 

The current analysis exploreeT the . relationships among the 

characteristics of a eurrieulum-based measurement system based on 

time-series analysis. In addition,- the relationship of such 

5 _ ^ -■ 

^h^acfteristics to student achievement in naturally occurring time- 
series data was investigated. 

The current results seem to indicate ^that the most distiihctive 
arid influential property of time-series data is the level of 
performance. The le\«el of performance factor accounts for 30% of the 
variance in the principal component factor analysis; this -figure is 
reminiscent of the 25^-40% of variance that Bloom (1976) suggested is 
accounted for by cognitive entry variables in any analysis enc 

» * 

.1 i IB " . 
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"achilvement. In addition, heavy loadinqs of the mean for the .Vear, as 
well as the Y-intercept (entry level), indiea^te that this factor 
remains important throughout the school year. .As Bloom (1975) had 
noted , cognitive, entry variabl es remain, unfortunately, ,the most 
powerful predictor of academic ^odteome." 

The*^ correlation of variability with / the - level of _ perfbrmahce 
fattor has interesting implications for time-series analysis. FtitUre 
investigations of time-series data may need to take into account the 
tendency of scores at a higher level t^ show greater vari abi 1 i ty. 
Since increased variation' has fcieeh shown to influence the accuracy of 
decisions made about time-series data,, these r^sult$ also may indicate 
that visual inference may be inherently more difficult at higher 
performance levels. These results are especially important in light 
of the common, fji.lure to take variabflity into account in visual 
analysis (Furlong & Wampold^ 1982).^ 

Number of changes in the student's instructional program 
correlated riegatively wi'th- both , the goal -meeting variables in Factor 
1^ and reading achievement. Since instructional interventions were 
intended to accelerate slopes' an^ increase the probability of meeting 
the lipng-range goal, these^ negative correlations are, at first glance, 
somewhat disturbing. ^ Given the small number of changes, made tn 
Students* instructional programs, however, it is likely that the 
correlations reflect the fact that changes were made drily, for students 
with extremely flat slopes. In order to evaluate the- effectiveness of 
'interventions, fu'^ther investigations will need to focus on changes in 
trend and variability resulting from individual interventions. 
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A positive eorrel at ibri between the steepness of the aimline 
initially set by the teacher arid the regression Slope based on actual 
student data iridicates that 'student time-series performance may in 
fact be influenced by teacher expectations. This correlation corifirms 

. . . _ . _ . _ i 

the tendency noted by the investigators for time-series data to follow 
the aimline, arid provides a cautidri against settirig goals at tod low a 
level.^ Yet^ the^e data do hot support- the thesis that setting goals 
higher wil 1 always accelerate student performance (Li rids ley, 1982) . 
The moderate negative correlation betweeri the number of times the qdal 
was met and the aiml ine/slope factor - indicates that settirig goals at 
too high a level may actually decrease student Success in reaching 
that goal. This conclusion is supported further by negative 
correlations between aimline and achievement measures: the steeper 
the aimline, the poorer the achievement. 

Similarly, excess variabil ity^often ',is regarded negatively in 
time-series analysis. In the current analysis, variability showed no 
relationship to slope, although it bo^je a moderate riegative 
relationship to the significance of the slope. Yet the staridard error' 
of ^estimate '%Vd' ndt explain a sigriifiearit probortiori df the 
achievement variarice iri any df the reqressiori arialyses. Thus, while 
variability may play a part iri determ^*ning the sigriificarice df trerids 
in the data payierh, ^it does ridt appear td be a useful- pre'dictdr df 
long-term outcomes. . , 

The number of data points per week appears to be a strong 

predictor of both the z-scdre of the slope, and performance on 

__ _ _ ._ __. __ . . ._. 

measures of reading achievement. A number of explanations cou^d be 
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offered. Increased measurement could simply afford increased reading 
practice; However^ given that this practice could amount ta a, maximum 
of 3 to 5 minutes per week, this seems relatively unlikely. A more 
plausible suggestion might be that i-ncreased measurement provided 
increased performance feedback to both the student and teacher. ,This 
feedback may in turn promote subtle instructional changes or increased^ 

motivation for students to achieve their goals in reading. 

i _ _ - 

Another strong predictor of reading achievement was the number of 
data points that exceeded the long-r^ge goal. The lohg-rahge goals 
were, for the most part, set so that students would reaqh their goal 
with an average of 1 to 2 words per' minute gain per week. These dat^ 
thus' provide some confirmation that thi^s rate of reading .growtli 
predicts positive achievement outcomes, and may well be an appropriate 
rate of reading growth for students in special settings. These 
findings also argue for the^ predictive validity of frequent; 
curriculum-based measurement, since success on the daily measures 
predicted success dh long-term achievement measures. 

It is interesting to compare the results of the current 
regression analyses with similar analyses perfbrmed on -this population 
using teacher effectiveness variables derived from preeess-prdduct 
research (cf. Skiba, Sevcik, Wesson, King, ^ Deno, 1983-^- Only one d^f 
the 12 independent variables investigated in that study. Frequency Wf 
Correct Answers, predicted • reading achievement at statistically 
significant levels, and the proportion of residual variance explained 
ranged from 1-35^. Both ^ the number of data points per week, and the 
number of times 'the goal was met, -explained a ^somewhat greater 

21 
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propdrtidh df residual variaricei ranging :^s hTgh as 28%. Variables 
such as academic engaged time, investigated in process-product 
research, have proven useful predictors of achievement in regular 
education; variables more directly related to monitoring student 
academic progress may prove to be \iore valuable in predicting outcomes^ 

. : _ ' ' \ 

in special education. _ . ^ 

<^ ^ 

These f i ndi ngs have important impl i cat ions in training - 

_ _ ___ __ ___ _^ ^_ _ V 

• practitioners in the use and interpretation of fneasurement systems 
based on time-series data. Fir'st, results indicate that properties of 
time-series data, such as level and variability are not independent of 
each other in nattj^lly occurring data. Thus, it may be necessary to 
trai^ practitioners to attend to interactions between time-series 
characteristics : when malting' judgments based dh visual inference. 
Second, in predicting long-ternf dutcdrnes, prdperties such as frequency 
df measurement may be as impdrtarit as ' more cdmmdnly attended to 
attributes, such as level and trend. Neither df these findings is 
particularly surprising, yet both represent important qualifications 
to the current methdddldgy df time-series analysis. 



18 » . i 

♦ _ _ • 

References o - 

BldbiTi, B. 5. HUmari characteristics and school learning . New York: 
McGraw-Rin;-1975. ^ , . . 

Bdri3,_W.' R. Jime and school learniYig... In C. Dehham & A. Lieberman^ 
Time {o learn . Washinqtnn, DC: National Institute of Education 

Denb, S. L.^ Mirlin, P. K., S Chiang, B . Identifying _va1 i d measures 
of reading. Except i-<yia1 Children. ^ 1982^.49, 36-47. 

- - _ - _ " _ i 

OePrdspero, A._, % Cohen,, S. _Ihconsistent-yisua1 analys^is of 

' intrasubiect data, i Journal of Applied BehavicH- Anal vsi s , 1979, 
12, 573-579. 

FurTdhg, M. _ J. , % Wampold, B. E. Intervention effects and relative 
variatidh as dimehsioh| in experts* use of visual analysis. 
Jdurnal df Applied Behavior Analysis , 1982, la, 415-421. 

Jones, R. R., Vaught, R. S., SWetnrott,M. Time series analysis in 
operant research. Journal af Ap^T- ied Behavior An a ly sJ-S^ , 1977, 
10, 151-166. . _ ^ 



Jones, R. R., Weinrott, M. R., S Vaught, R. S. Effects of serial 
dependency on the agreement between visual and statistical 
inference. Journal of Appi lecL Behavior Analysis , 1978, 11 , 
272-233. 

. ___ _ - 

Karlsen, B., Madden^ R., & Gardner, ,E. F. Stanford dfagndstic 

reading test . New York: Harcourt Brace Jovandvich, 1976. ^ 

kazdin,. A. E. Statistical analyses for sing1e-case_exp^rimehta1 
design. In M. Hersen & B. Barlow (Eds. ), "^ Single erase 
exper-i mental designs: Strategies for studying Hehavjdr change . 
New York: Pergamon Press, 1975. 

kratochwin, T,_R._ Foundations of time-spies' research, til T. R. 
Kratochwill (Ed.), Single sab^'ect research: Strategies for 
evaluating change . New York: Academic Press, 1978. 

Lindsle^^, 0. R. Hdw teachers cduTd be rewarded for student learning. 
Paper presented at_ the Twelfth Annual Irivitatignal Conference . 
dn Leadership in Special Education, Minneapolis, November, 1982. 

• y _ '_ 

Marstdh, D., Ldwry, L., Oeno, S., ^ Mirkin^.P. An analysis a £ 
learning trends in simple iijeasures of rea ding, spe lX i ng, 
Written expression: A longitudinal study (Research Report No. 
49) . Minneapol is: tJiri versi'ty df Minnesota, ^Institute for 
Research on Learning Disabil ities, 1981 . 



_ . . * 

Michael, 0._ Statistical inference for individual organism research: 
Mixed blessing or curse? Journal of Applied Behavior Analysis , 
* 1974, i, 647-653. 

Mirkin, P., Deno, ST^ Fuchs, U, Wessbh,C.,^Tinda1 , G.^ Marstoh, f)., 
& Kuehnle*K. Procediires^o develop and monitor progress—oa 
lEP goals/ Minneapol is:^ University of*>1innesota, Insti tute 
for Research on Learning Disabilities, 1981.. 

Parsonson, B. S., & Baer, 9. M. The analysis and presentation of 
graphic data. In T. R. Kratodhwill (Ed.), Single subj-ect 
research: Strategies for evaluating change . New York: 
^ Aeadqfiic Press, 1978i 

Skiba, R., Marston, D., Wesson, r.^^ Seveik, 8., % Deno;, S. b. ^: 

Characteristics of the time-series data collected through . 

curriculum-based reading measurement (Reseajrch Report No. 125}^. 
Minneapol is;_ Universi of .Minnesota, Institute for Research 
pn Learning Disabilities; 1983. 

Skiba, Ri, Seveik^ B., Wess^on, C., King, R. , 'Der>d, S. The 

noh'reffect of processrproduct variables thre^durce classrooms 
(Research Report No. 121)^ Minneapol is: University of _ ~7_ __ 
Minnesota, T/istitute for Research on Learning Hisabil ities, 1983 

Tiridal, G., Deno, S. L.,_% YsseVdyke, H. E. Visual anal ysis of . * 
' time-series data:. Factors of influence and level of reliab ility 
iResearch Report No. 112V. Minneapolis: ^ University of 
Minnesota, Institute for Research on Leariiing Disabil ities, 1983 

Tryon, W. W. A simpM^led time-series an^alysis fbr evaluating' 

treatment intervenUons . Journal of Appl ied Behavi^jr^ Anal ys^i s , . 
1982^ 15, 423-429. 

White, 0» R., % Haring, 'E. G. Excj^t^waX-te ach i nq " ( 2nd ed.V. 
Columbus, OH:: Charles E. Merrill, 19^0. 




i ■ , • 

Footnotes 

The autHors gratefully acknowledge the cdri'tri butjoh? of Or. 

Gerald Tindal in directing our attention to the c-statistic as a 
method of time-series analysis; • - 

Hhe actual calculations involved in the (^statistic, as cited by 

Tryon' (i982), are: ^ . ^; 

' ■ i'l (^i - ^i.l^' 



C = 1 



ZZ (X. - X)^ 



i^l , ' 

where the numerator Of the right hand term is the sum of the fN-1) 

squarpd consecutive differences associated with the time- series. The 

denhminator is twice the sum of .he (N) squared deviations Of the 

time-series data points from their norm. 

^ ^The standard error of the c-statistic is 

. . 

_ /" N+2 * ■ • 

^ ^/ .(N-JHN + 1) 

the c-stat'istic may be converted to a z-statistic and testeH for 

ST:gnif icanee through* the fbHowi^tj ratio: 

7 = _ - 
Sc ^_ • ^ 
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■ Table 1 ^.^ ■ ^ 
Distribution Of Students by Grade tevel 

- ' » ■ 



Number of Students 




2 
18 
15 
16 
12 

3 
■ 2 



i 



Percentage 



2.9 
.26.5 
22.1 
23.5 

17:6 
4:4 
2.9 
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Table 2 

Bescri jjtive -Statistics for Time-Series Variables 







Mea nr-s.^ 

X 






Y-ihtercep-t 


37.01* ■ , 


11 .25, 




Mean level for the yelr 


51 . 03* 


11; 67 




■ - - _ _ _ _ 
Standard error of estimate 




4 ?fl 

H . CO 




Slope 


1 ..Be** ' 


. 87 . 




Aiml ihe 


1.68** 


■ .53 

* • . ■ 




Number of data points/week • 


2.59 


.67 




Number of in struct ional interventions 


.67 


1.T4 




for the year r . 








Z-score of the C-statistic 


3 . 32 


2^43 




Number of times goal level was 


8.54 


9.26 




exceeded 









Frequency Percent al 

Students reachirig iohg-rahge goal 58 85.3? 

at 1 east once 

*Expressed in words per minute 
**Expressed in words per minute gained per week 
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• Table 3 

Results of Principal Cdrtiponerits Factor* .Analysis 



Variabl es^ 




Factor 1^ 


Factor 2 


Factor 3 


Y-INT 






-.4093- 


-.0760 


MY 




.8802 


-.0689 


.1592 • 


SEE 




.5220 


-.4026 


-.1259 


SLOPE 




■ ' .2778 * 


.3362 


n -i633 


AiriLINE 




-.0053 


' -.1265 


.7565 


DATA PTS 




• -.0087 


.7380 


-;10O1 


.CHANGE 




-.4463 


-;0731 


' -;1653 


Z-S.edRE 




-;12a4 


;9324 


.0539 


GQAt 




.5460' 


.0242 


-.0192 


NSBAb 




^ .6215 

-r^. ^ 


.4444 


-.3235 


^Y-INT > Y- 
HY^ Mean 


■intercept 
leveT of- 


performance for the 


year 





SEE_.= Standard error of estimate of the regression slope 
SLOPE. = Regression slope 

AIMLINE = The trend of the aimlirie drawn by the teacher between the 

Y^intercept and the long-range goal level 
* DATA PTS =_ Number ofdata points per week ^ 
CHANGE ^ Number of i hstructibhal intervehtidhs over the course of 

the school year - 
ZSCORE = Theu^-score of the c-statistic (see Footnote Ij 
GOAL = Binary variable indicating whether the goal. was met at least 

once 

NGOAL = The number _^f times the long-range^ goal level was exceeded 
in the time-serie?^ » . ; (_( 

'^Variables Ibadirig positively or negatively on the factor are underlined. 
The level chosen for analysis was ^ .30. 
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Tabl e 4 

SuiTinary of Regression of Time-Series Characteristics 6n Achlevenierit Measures 



Propdrtldn- 
of Residual 
Variance 



Proportion of 
Variance Accounted 

for t)y Prp^f st 

Achievement 



Independent 
Var labl es with 
?''§n''f leant ^ 
Beta-weights 



Sign 



Proportion of 
Residual Variance -- 
^ccpuntcd for by^ Al 1 
Independent Variable! 



NGOAL***' 


[+5 


.18 


.29 


AiMtlNE** 


(-) 


;04 




CHANGE** 




.09 


•21 


eoAt* 


(+) 


.04 










.05 


CHANGE** 


(-) 


.05 


.08 


NGOAL*** 


(0 


.28 


;56 


yt-INT** 

'OATAPTS* 


{+) 


.11 




(+) 


.04 




AIMLINE* 


(-) 


.04 




MEAN/YEAR* 


(+) 


.04 




DATAPTS*** 


(+j 


\.23 


.35 



Passage Score - Time 3 
(N-56) 

SORT - Comprehensions 
Subtests (N = 57) ^ 

SORT - Structural Analysis 
Subtests (N=57) 

SORT - Total Score 

Passage Gain Scdre*^ 
(N=55) 



Percent Gain 
(N=55) 



.32 
.20 
.35 
.35 



^The passage score at time 1 was used to control for entering achievement. 

^Gain in words read per minute i^rom the* third grade passages from time 1 (October) to time 3 (May). 

^OATAPTS = Number of data points per week 
NGpAL = Number of times goal wa^ 

CHANGE = Number of changes made in the instructional program 
MEAN/yEAR= Mean level of performance for the year _ 

AIMLINE = Steepness, in words per week, of the aimlihe drawn by the teacher 
Y-INT = Y-intercept. 

^The_ proportion of the residual variance accounted for by the variable listed in Column 2, where 
residaal variance refers to the variance ^remaining In the post-achievement m^easure after entering 
achievement at time 1. 



The proportion of the residual variance accounted for by aH independent variables after accounting 
for pre^achievement. ' - . 

*P f .10 : 
**p < .05 
;***p < .001 
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Student: 



Gr/Agg' 5/10 $ch: ail Ixiv. Groove Tch: Ms, winkler Academic Area: Reading 
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Fir "-$)\ Example Of Individual Time-Series Data. 
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LRG: 




Condition 



In — weeks, when 

(total # weeks) 
presented with stories from 

bevel , __ ^ 

(reading series) , 




PC 



Behavior 



Criteria 



student wil 1 
read aloud 



at the rate of 50 

wpm or better 

5 or fewer errors , 



Figure 2, Format for Ldhg-Range Goal: Reading 
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Condi tibh 



STO ^ Each stccessive week^ when 
presented with a raridbm 
selection from Level 



of 



(reading series 



77 



Behavior 



student wil 1 
read aloud 



Criteria 



at ah average 
Increase of 



(repeated-actual 
performance/total # 
weeks) words correct/ 
minute and no increase 
in errors; q- '-- 



Figure 3. Forijat for Short-Term Objective: Reading 
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